Interior Camera System for a Self Driving Car

ABSTRACT

The technology provides an interior camera sensing system for self-driving vehicles. The sensor system includes image sensors and infrared illuminators to see the vehicle&#39;s cabin and storage areas in all ambient lighting conditions. The system can monitor the vehicle for safety purposes, to detect the cleanliness of the cabin and storage areas, as well as to detect whether packages or other objects have been inadvertently left in the vehicle. The cameras are arranged to focus on selected regions in the vehicle cabin and the system carries out certain actions in response to information evaluated for those regions. The interior space is divided into multiple zones assigned different coverage priorities. Regardless of vehicle size or configuration, certain actions are performed according to various ride checklists and the imagery detected by the interior cameras. The checklists include pre-ride, mid-ride, and post-ride checklists.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 16/718,462, filed Dec. 18, 2019, the entire disclosure of whichis incorporated herein by reference.

BACKGROUND

Autonomous vehicles, such as vehicles that do not require a humandriver, can be used to aid in the transport of passengers from onelocation to another. Such vehicles may operate in a fully autonomousmode without a person providing driving input, or in a partiallyautonomous mode with a driver having control over one or more aspects ofvehicle operation.

BRIEF SUMMARY

The technology relates to an interior sensor system for use withvehicles that can operate in an autonomous driving mode. The sensorsystem includes visible and near infrared (IR) sensors to see thevehicle's cabin in all ambient lighting conditions. According to certainaspects, the system can be used to monitor occupants for safetypurposes, including whether seatbelts are being worn properly. It canalso be used to detect the cleanliness of the cabin and storage areas,for instance to schedule maintenance or service should the vehicle needto be cleaned. In addition to this, the in-vehicle sensor system can beused for rider support, including providing a videoconference capabilitywith a remote assistance service, as well as to detect whether packagesor other objects have been inadvertently left in the vehicle, whether aperson is near the open trunk before closing the trunk lid, whether apassenger or pet has a hand, paw or other appendage outside an openwindow, etc.

According to one aspect, an interior sensing system for a vehicleconfigured to operate in an autonomous driving mode is provided. Theinterior sensing system includes a plurality of image sensors disposedalong different surfaces within an interior area of the vehicle. Theplurality of image sensors are configured to capture images of theinterior area of the vehicle. The system also includes one or moreinfrared units configured to illuminate one or more zones within thevehicle during image capture by the plurality of image sensors, as wellas a control system operatively coupled to the plurality of imagesensors and the one or more infrared units. The control system includesone or more processors configured to initiate image capture by theplurality of image sensors in accordance with a predetermined ridechecklist, receive captured imagery from the plurality of image sensorsin response to the predetermined ride checklist, perform processing ofthe received imagery to detect a current status of the interior area ofthe vehicle based on detected objects or conditions within the vehicle,and cause one or more systems of the vehicle to perform an action basedon the current status of the interior area of the vehicle. Thepredetermined ride checklist may be selected from a pre-ride checklist,a mid-ride checklist, and a post-ride checklist based on a current ridestatus.

The control system may be further configured to determine an amount ofambient light, and to actuate the one or more infrared units in responseto the determined amount of ambient light. Here, when the predeterminedride checklist is a mid-ride checklist and the determined amount ofambient light is below a threshold level, the control system may beconfigured to prevent illumination by one or more interior visible cabinlights.

In one scenario, the plurality of image sensors is disposed alongdifferent surfaces within the interior area of the vehicle to obtainimagery from a plurality of zones within the vehicle. The plurality ofzones may be ranked according to multiple priority levels. At least oneof an image capture rate and an image resolution in each of theplurality of zones may be based on the priority level corresponding tothe zones. A field of view for a given one of the plurality of imagesensors focused on a selected zone may be based on the priority levelfor the selected zone.

In another scenario, the predetermined ride checklist is a pre-ridechecklist and the current status indicates that there is an object inthe vehicle or service is required. Here, the control system isconfigured to cause a communication system of the vehicle to perform theaction of contacting a remote facility to service the vehicle or removethe object. Contacting the remote facility to service the vehicle mayinclude a request to either clean the vehicle or to reset a vehiclecomponent to a default setting.

In a further scenario, when the predetermined ride checklist is apost-ride checklist and the current status indicates that there is anobject in the vehicle from a prior passenger, the control system isconfigured to cause a communication system of the vehicle to contact aremote facility to remove the object from the vehicle, contact the priorpassenger regarding the object in the vehicle, or contact both theremote facility and the prior passenger regarding the object in thevehicle. Contacting the prior passenger may include informing the priorpassenger what type of object was left in the vehicle and where theobject can be retrieved.

In an example, causing one or more systems of the vehicle to perform theaction based on the current status of the interior area of the vehicleincludes causing a planning system of the vehicle to plan a change inroute. Each of the one or more infrared units may be collocated with acorresponding one of the plurality of image sensors. Alternatively, eachof the one or more infrared units may be located separately from theplurality of image sensors and arranged to avoid flare during imagecapture by the plurality of image sensors. At least one of the one ormore infrared units may be further configured to provide illumination inat least part of the visible light spectrum.

In another example, initiation of the image capture by at least one ofthe plurality of image sensors includes performing high dynamic rangeimage capture using a plurality of different exposure levels. Andprocessing of the received imagery to detect the current status mayinclude the one or more processors performing a machine learningoperation to determine whether a given one of the plurality of imagesensors requires cleaning or if there is an obstruction of the givenimage sensor.

According to another aspect, a method of controlling an interior sensingsystem for a vehicle configured to operate in an autonomous driving modeis provided. The method comprises initiating, by one or more processors,image capture by the plurality of image sensors in accordance with apredetermined ride checklist, the predetermined ride checklist beingstored in memory of a control system of the vehicle; in response to thepredetermined ride checklist, the one or more processors receivingcaptured imagery from a plurality of image sensors disposed alongdifferent surfaces within an interior area of the vehicle, the pluralityof image sensors being configured to capture images of the interior areaof the vehicle; the one or more processors performing processing of thereceived imagery to detect a current status of the interior area of thevehicle based on detected objects or conditions within the vehicle; andthe one or more processors causing one or more systems of the vehicle toperform an action based on the current status of the interior area ofthe vehicle.

In one example, the method further comprises determining an amount ofambient light; actuating one or more infrared units disposed within theinterior area of the vehicle in response to determining the amount ofambient light; and when the predetermined ride checklist is a mid-ridechecklist and the determined amount of ambient light is below athreshold level, preventing illumination by one or more interior visiblecabin lights.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-B illustrate an example passenger-type vehicle configured foruse with aspects of the technology.

FIG. 2 illustrates in interior view of the vehicle of FIGS. 1A-B inaccordance with aspects of the technology.

FIG. 3 is a block diagram of systems of an example vehicle in accordancewith aspects of the technology.

FIGS. 4A-B illustrate example views of interior sections of a vehicle inaccordance with aspects of the technology.

FIGS. 5A-F illustrate vehicle regions and priorities in accordance withaspects of the technology.

FIGS. 6A-B illustrate an example of a pre-ride check in accordance withaspects of the technology.

FIGS. 7A-B illustrates an example of a mid-ride check in accordance withaspects of the technology.

FIGS. 8A-B illustrates an example system in accordance with aspects ofthe technology.

FIG. 9 illustrates an example method in accordance with aspects of thetechnology.

DETAILED DESCRIPTION

The interior camera system according to the present technology is ableto focus on selected regions in the vehicle cabin and carry out certainactions in response to information evaluated for those regions. In anideal arrangement, the sensor system would have fields of view (FOV)capable of covering all interior cabin areas. However, this may not befeasible due to cost, complexity or other constraints. Thus, the cabincan be divided into multiple zones, where the zones may be assigneddifferent coverage priorities. The number of zones that may be employedcan depend on the type of vehicle. For instance, a minivan may have morezones than a two-door sports car because it has an additional row ofseats and/or additional storage areas. Regardless of the vehicle size orconfiguration, there are certain threshold requirements to be met oroperations to perform when the vehicle is on the way to pick up apassenger, when the passenger is being driven to a destination, and oncethe passenger exists the vehicle. These and other features are discussedbelow.

Example Vehicle Systems

FIG. 1A illustrates a perspective view of an example passenger vehicle100, such as a minivan, sport utility vehicle (SUV) or other vehicle.FIG. 1B illustrates a top-down view of the passenger vehicle 100. Asshown, the passenger vehicle 100 includes various external sensors forobtaining information about the vehicle's outside environment, whichenable the vehicle to operate in an autonomous driving mode. Forinstance, a roof-top housing 102 may include a lidar sensor as well asvarious cameras, radar units, infrared and/or acoustical sensors.Housing 104, located at the front end of vehicle 100, and housings 106a, 106 b on the driver's and passenger's sides of the vehicle, may eachincorporate lidar, radar, camera and/or other sensors. For example,housing 106 a may be located in front of the driver's side door along aquarter panel of the vehicle. As shown, the passenger vehicle 100 alsoincludes housings 108 a, 108 b for radar units, lidar and/or camerasalso located towards the rear roof portion of the vehicle. Additionallidar, radar units and/or cameras (not shown) may be located at otherplaces along the vehicle 100. For instance, arrow 110 indicates that asensor unit (112 in FIG. 1B) may be positioned along the rear of thevehicle 100, such as on or adjacent to the bumper. And arrow 114indicates a series of sensor units 116 arranged along a forward-facingdirection of the vehicle. In some examples, the passenger vehicle 100also may include various sensors for obtaining information about thevehicle's interior spaces (not shown).

By way of example, each external sensor unit may include one or moresensors, such as lidar, radar, camera (e.g., optical or infrared),acoustical (e.g., microphone or sonar-type sensor), inertial (e.g.,accelerometer, gyroscope, etc.) or other sensors (e.g., positioningsensors such as GPS sensors). While certain aspects of the disclosuremay be particularly useful in connection with specific types ofvehicles, the vehicle may be any type of vehicle including, but notlimited to, cars, trucks, motorcycles, buses, recreational vehicles,etc.

There are different degrees of autonomy that may occur for a vehicleoperating in a partially or fully autonomous driving mode. The U.S.National Highway Traffic Safety Administration and the Society ofAutomotive Engineers have identified different levels to indicate howmuch, or how little, the vehicle controls the driving. For instance,Level 0 has no automation and the driver makes all driving-relateddecisions. The lowest semi-autonomous mode, Level 1, includes some driveassistance such as cruise control. Level 2 has partial automation ofcertain driving operations, while Level 3 involves conditionalautomation that can enable a person in the driver's seat to take controlas warranted. In contrast, Level 4 is a high automation level where thevehicle is able to drive fully autonomously without human assistance inselect conditions. And Level 5 is a fully autonomous mode in which thevehicle is able to drive without assistance in all situations. Thearchitectures, components, systems and methods described herein canfunction in any of the semi or fully-autonomous modes, e.g., Levels 1-5,which are referred to herein as autonomous driving modes. Thus,reference to an autonomous driving mode includes both partial and fullautonomy.

Turning to FIG. 2, this figure illustrates an example view 200 withinthe cabin of the vehicle 100, for instance as seen from the front seats.In this view, a dashboard or console area 202 which includes an internalelectronic display 204 is visible. Although vehicle 100 includes asteering wheel, gas (acceleration) pedal, or brake (deceleration) pedalwhich would allow for a semiautonomous or manual driving mode where apassenger would directly control the steering, acceleration and/ordeceleration of the vehicle via the drivetrain, these inputs are notnecessary for a fully autonomous driving mode. Rather, passenger inputmay be provided by interaction with the vehicle's user interface systemand/or a wireless network connection for an app on the passenger'smobile phone or other personal computing device. By way of example, theinternal electronic display 204 may include a touch screen or other userinput device for entering information by a passenger such as adestination, etc. Alternatively, internal electronic display 204 maymerely provide information to the passenger(s) and need not include atouch screen or other interface for user input.

Also shown in FIG. 2 are sensor units 206, 208 and 210, which may belocated at different places within the cabin and storage areas. Thesensor units may each include one or more cameras (e.g., optical and/orIR imaging devices, high dynamic range (HDR)-capable cameras, etc.),time of flight sensors, IR illuminators, microphones, pressure sensorsor other sensors. Information obtained from the sensors can be sent toan onboard computer system for processing and/or analysis in real timeto determine, e.g., a state of the vehicle include seating locations forany passengers, whether there are packages or other objects on a seat,on the floor, in a cupholder, etc. The obtained information may be firstprocessed partially or fully by the sensor unit, or raw or partiallyprocessed data may be sent to the onboard computer system. For instance,an HDR camera captures multiple images having different exposure levelsat the same time (e.g., 2-4 images). The camera sensor unit maypre-process the images to obtain a single image that is then sent to thecomputer system for further image processing. Based on this informationthe vehicle may take certain actions or communicate with a remote systemthat can schedule a service or provide assistance. By way of example, aneural network evaluating the camera images may determine whether thecamera requires cleaning or if there is an obstruction of the camera. Inthe first situation, the computer system may actuate a cleaning elementsuch as a wiper to clean the camera lens. In the second situation, aservice call may be required to correct the obstruction.

FIG. 3 illustrates a block diagram 300 with various components andsystems of an exemplary vehicle, such as passenger vehicle 100, tooperate in an autonomous driving mode. As shown, the block diagram 300includes one or more computing devices 302, such as computing devicescontaining one or more processors 304, memory 306 and other componentstypically present in general purpose computing devices. The memory 306stores information accessible by the one or more processors 304,including instructions 308 and data 310 that may be executed orotherwise used by the processor(s) 304. The computing system may controloverall operation of the vehicle when operating in an autonomous drivingmode.

The memory 306 stores information accessible by the processors 304,including instructions 308 and data 310 that may be executed orotherwise used by the processors 304. The memory 306 may be of any typecapable of storing information accessible by the processor, including acomputing device-readable medium. The memory is a non-transitory mediumsuch as a hard-drive, memory card, optical disk, solid-state, etc.Systems may include different combinations of the foregoing, wherebydifferent portions of the instructions and data are stored on differenttypes of media.

The instructions 308 may be any set of instructions to be executeddirectly (such as machine code) or indirectly (such as scripts) by theprocessor(s). For example, the instructions may be stored as computingdevice code on the computing device-readable medium. In that regard, theterms “instructions”, “modules” and “programs” may be usedinterchangeably herein. The instructions may be stored in object codeformat for direct processing by the processor, or in any other computingdevice language including scripts or collections of independent sourcecode modules that are interpreted on demand or compiled in advance. Thedata 310 may be retrieved, stored or modified by one or more processors304 in accordance with the instructions 308. In one example, some or allof the memory 306 may be an event data recorder or other secure datastorage system configured to store vehicle diagnostics and/or obtainedsensor data, which may be on board the vehicle or remote, depending onthe implementation. The data may include, for instance, inspection oroperating checklists or other use cases that can be used pre-ride,in-ride and/or post-ride. The data may also include training sets,object models or other information to perform object recognition fordifferent types of objects (e.g., passengers, pets or service animals,bags or other packages, mobile phones or other personal computerdevices, etc.)

The processors 304 may be any conventional processors, such ascommercially available CPUs. Alternatively, each processor may be adedicated device such as an ASIC or other hardware-based processor.Although FIG. 3 functionally illustrates the processors, memory, andother elements of computing devices 302 as being within the same block,such devices may actually include multiple processors, computingdevices, or memories that may or may not be stored within the samephysical housing. Similarly, the memory 306 may be a hard drive or otherstorage media located in a housing different from that of theprocessor(s) 304. Accordingly, references to a processor or computingdevice will be understood to include references to a collection ofprocessors or computing devices or memories that may or may not operatein parallel.

In one example, the computing devices 302 may form an autonomous drivingcomputing system incorporated into vehicle 100. The autonomous drivingcomputing system is configured to communicate with various components ofthe vehicle. For example, the computing devices 302 may be incommunication with various systems of the vehicle, including a drivingsystem including a deceleration system 312 (for controlling braking ofthe vehicle), acceleration system 314 (for controlling acceleration ofthe vehicle), steering system 316 (for controlling the orientation ofthe wheels and direction of the vehicle), signaling system 318 (forcontrolling turn signals), navigation system 320 (for navigating thevehicle to a location or around objects) and a positioning system 322(for determining the position of the vehicle, e.g., including thevehicle's pose). The autonomous driving computing system may employ aplanner module 324, in accordance with the navigation system 320, thepositioning system 322 and/or other components of the system, e.g., fordetermining a route from a starting point to a destination or for makingmodifications to various driving aspects in view of current or expectedtraction conditions.

The computing devices 302 are also operatively coupled to a perceptionsystem 326 (for detecting objects in the vehicle's internal and externalenvironments), a power system 328 (for example, a battery and/or gas ordiesel powered engine) and a transmission system 332 in order to controlthe movement, speed, etc., of the vehicle in accordance with theinstructions 308 of memory 306 in an autonomous driving mode which doesnot require or need continuous or periodic input from a passenger of thevehicle. Some or all of the wheels/tires 330 are coupled to thetransmission system 332, and the computing devices 32 may be able toreceive information about tire pressure, balance and other factors thatmay impact driving in an autonomous mode.

The computing devices 302 may control the direction and speed of thevehicle, e.g., via the planner module 324, by controlling variouscomponents. By way of example, computing devices 302 may navigate thevehicle to a destination location completely autonomously using datafrom the map information and navigation system 320. Computing devices302 may use the positioning system 322 to determine the vehicle'slocation and the perception system 326 to detect and respond to objectswhen needed to reach the location safely. In order to do so, computingdevices 302 may cause the vehicle to accelerate (e.g., by increasingfuel or other energy provided to the engine by acceleration system 314),decelerate (e.g., by decreasing the fuel supplied to the engine,changing gears, and/or by applying brakes by deceleration system 312),change direction (e.g., by turning the front or other wheels of vehicle100 by steering system 316), and signal such changes (e.g., by lightingturn signals of signaling system 318). Thus, the acceleration system 314and deceleration system 312 may be a part of a drivetrain or other typeof transmission system 332 that includes various components between anengine of the vehicle and the wheels of the vehicle. Again, bycontrolling these systems, computing devices 302 may also control thetransmission system 332 of the vehicle in order to maneuver the vehicleautonomously.

Navigation system 320 may be used by computing devices 302 in order todetermine and follow a route to a location. In this regard, thenavigation system 320 and/or memory 306 may store map information, e.g.,highly detailed maps that computing devices 302 can use to navigate orcontrol the vehicle. As an example, these maps may identify the shapeand elevation of roadways, lane markers, intersections, crosswalks,speed limits, traffic signal lights, buildings, signs, real time trafficinformation, vegetation, or other such objects and information. The lanemarkers may include features such as solid or broken double or singlelane lines, solid or broken lane lines, reflectors, etc. A given lanemay be associated with left and/or right lane lines or other lanemarkers that define the boundary of the lane. Thus, most lanes may bebounded by a left edge of one lane line and a right edge of another laneline.

By way of example only, the perception system 326 may include one ormore light detection and ranging (lidar) sensors, radar units, cameras(e.g., optical and/or IR imaging devices, with or without aneutral-density filter (ND) filter), positioning sensors (e.g.,gyroscopes, accelerometers and/or other inertial components), acousticalsensors (e.g., microphones or sonar transducers), and/or any otherdetection devices. In addition, IR illuminators may also be employed inconjunction with the cameras, for instance to illuminate within thecabin or immediately adjacent to the vehicle. Such sensors of theperception system 326 may detect objects outside of the vehicle andtheir characteristics such as location, orientation, size, shape, type(for instance, vehicle, pedestrian, bicyclist, etc.), heading, speed ofmovement relative to the vehicle, etc. To aid in the detection andclassification of objects, one or more illuminators may have acombination of IR and visible light to obtain certain color information.Thus, the illuminator may be configured to also emit light in at leastpart of the visible light spectrum.

As shown in FIG. 3, the perception system 326 includes one or moreexternal sensors 334 for detecting objects external to the vehicle. Thesensors 334 are located in one or more sensor units around the vehicle.The detected objects may be other vehicles, obstacles in the roadway,traffic signals, signs, trees, bicyclists, pedestrians, etc. The sensors334 may also detect certain aspects of weather or other environmentalconditions, such as snow, rain or water spray, or puddles, ice or othermaterials on the roadway.

The perception system 326 also includes other sensors 336 within thevehicle to detect objects and conditions within the vehicle, such as inthe passenger compartment and trunk region. For instance, such sensorsmay detect, e.g., one or more persons, pets, packages, etc., as well asconditions within and/or outside the vehicle such as temperature,humidity, etc. This can include detecting where the passenger(s) issitting within the vehicle (e.g., front passenger seat versus second orthird row seat, left side of the vehicle versus the right side, etc.).The interior sensors 336 may detect the proximity, position and/or lineof sight of the passengers in relation to one or more display devices ofthe passenger compartment, for example to determine how best to presentinformation to the passengers during a ride.

The raw data obtained by the sensors can be processed by the perceptionsystem 336 and/or sent for further processing to the computing devices302 periodically or continuously as the data is generated by theperception system 336. Computing devices 302 may use the positioningsystem 322 to determine the vehicle's location and perception system 326to detect and respond to objects when needed to reach the locationsafely, e.g., via adjustments made by planner module 324. In addition,the computing devices 302 may perform calibration of individual sensors,all sensors in a particular sensor assembly, or between sensors indifferent sensor assemblies or other physical housings.

As illustrated in FIGS. 1A-B and 2, certain sensors of the perceptionsystem 326 may be incorporated into one or more sensor assemblies orhousings outside and inside the vehicle. In one example, these may beintegrated into the side-view mirrors on the vehicle. In anotherexample, other sensors may be part of the roof-top housing 102, or othersensor housings or units 106 a,b, 108 a,b, 112 and/or 116. In furtherexamples, cameras and IR illuminators may be disposed within the vehiclealong the headliner, the A, B, C or D pillars, etc. The computingdevices 302 may communicate with the sensor assemblies located on orotherwise distributed along the vehicle. Each assembly may have one ormore types of sensors such as those described above.

Returning to FIG. 3, computing devices 302 may include all of thecomponents normally used in connection with a computing device such asthe processor and memory described above as well as a user interfacesubsystem 338. The user interface subsystem 338 may include one or moreuser inputs 340 (e.g., a mouse, keyboard, touch screen and/ormicrophone) and one or more display devices 342 (e.g., a monitor havinga screen or any other electrical device that is operable to displayinformation). In this regard, an internal electronic display may belocated within a cabin of the vehicle (e.g., 204 in FIG. 2) and may beused by computing devices 302 to provide information to passengerswithin the vehicle. By way of example, displays may be located, e.g.,along the dashboard, on the rear of the front row of seats, on a centerconsole between the front row seats, along the doors of the vehicle,extending from an armrest, etc. Other output devices, such as speaker(s)344 may also be located within the passenger vehicle. The passenger(s)may communication directly with the vehicle via one or more deviceinputs 346. The inputs 346 may include a touch screen on an internalelectronic display, a microphone for receiving spoken instructions, ahaptic sensor for receiving physical feedback, etc.

The vehicle also includes a communication system 348. For instance, thecommunication system 348 may also include one or more wirelessconfigurations to facilitate communication with other computing devices,such as passenger computing devices within the vehicle, computingdevices external to the vehicle such as in another nearby vehicle on theroadway, and/or a remote server system. The network connections mayinclude short range communication protocols such as Bluetooth™,Bluetooth™ low energy (LE), cellular connections, as well as variousconfigurations and protocols including the Internet, World Wide Web,intranets, virtual private networks, wide area networks, local networks,private networks using communication protocols proprietary to one ormore companies, Ethernet, WiFi and HTTP, and various combinations of theforegoing. The communication system 348 may thus include one or moreantennas located within the cabin and/or on the vehicle's roof, as wellas one or more transceiver modules coupled to the antennas for providingwireless communication.

While the components and systems of FIG. 3 are generally described inrelation to a passenger vehicle arrangement, as noted above thetechnology may be employed with other types of vehicles, such as buses,campers, cargo vehicles, etc. In larger vehicles, the user interfaceelements such as displays, microphones and speakers may be distributedso that each passenger has his or her own information presentation unitand/or one or more common units that can present status information tolarger groups of passengers. Similarly, the interior sensors 336 of theperception system 326 may be arranged (e.g., based on one or morecoverage priorities) to have fields of view encompassing different areasthroughout the vehicle.

Example Implementations

In view of the structures and configurations described above andillustrated in the figures, various aspects will now be described inaccordance with aspects of the technology.

A self-driving vehicle, such as a vehicle with level 4 or level 5autonomy that can perform driving actions without human operation, hasunique requirements and capabilities. This includes making drivingdecisions based on a planned route, received traffic information, andobjects in the external environment detected by the onboard sensors. Italso includes determining a status of the vehicle before picking up apassenger, while transporting the passenger to his or her destination,and after the passenger exists the vehicle. In such situations, in orderto determine the vehicle status and operate accordingly, the vehicle mayrely on sensor data obtained from interior sensors distributedthroughout the vehicle, such as in the passenger compartment and trunk.

FIG. 4A illustrates a top-down view 400 of a vehicle cabin of an examplevehicle, such as vehicle 100 of FIG. 1. As shown, the cabin includes afront seat area 402 and a rear seat area 404. A center console 406 maybe disposed between the front seats. FIG. 4B illustrates a perspectiveview 410 of a trunk region or other cargo space 412 of the vehicle. Asnoted above, ideally the interior sensor system may have sensors withfields of view that encompass all areas within the vehicle. However,this may not be feasible due to the number of sensors that would berequired, as well as the cost of deploying and servicing such sensors.Thus, according to one aspect of the technology, the interior sensorsystem employs multiple priority levels to cover different regions ofthe vehicle.

By way of example, there may be three priority levels generallycategorized as follows: high priority, medium priority, and lowpriority. In this example, high priority areas may encompass, e.g., allseat surface (including both lower and upper surfaces), the centerconsole area, the trunk floor surface, and areas adjacent to the trunksuch as where a person or their legs would be when leaning into thetrunk. The medium priority areas may encompass front and rear floorsurfaces. And low priority areas may include storage locations, such asin-door receptacles, seatback compartments, etc. Other scenarios mayhave two, four or more priority levels.

The priority level for a given area may be used to select the refreshrate and/or resolution of the imagery obtained for that area. By way ofexample only, a high priority area may have a refresh rate on the orderof 0.5-5.0 seconds at a maximum resolution of the imaging device, whilea low priority area may have a refresh rate on the order of 30-60seconds at a lower resolution (e.g., 25-50% of the maximum resolution).A medium priority area may have a refresh rate on the order of 3.0-45seconds at a medium resolution (e.g., 70-80% of the maximum resolution).These refresh and resolution rates are merely exemplary, and may belower or higher. In additional or alternatively, the image processingtechniques used for the imagery may be tailored to the priority level,region of the vehicle, or other criteria. For instance, for seat areas,the image processing may use training data or other machine learningapproaches (e.g., a neural network) that are primarily focused onrecognizing people, whereas for storage areas the system may be moretuned to detecting the type of object (e.g., a package, groceries,etc.). There may be different precision recall requirements fordifferent priorities, thereby optimizing compute for the on-boardprocessing system. Alternatively or additionally, based on the differentpriorities the system could also trigger a request to a remote acustomer service group. For instance, if the priority level meets orexceeds a certain threshold, or when there is uncertainty about theimage (e.g., on-board processing cannot determine if there is a leftitem or a mess, or if there is an occupant), the request for remoteassistance may be sent. This can include the remote system performingadditional object recognition processing on the image of interest orhaving a remote assistance technician review the image.

FIGS. 5A-F illustrate different regions of the vehicle cabin and trunkfor an exemplary vehicle, such as vehicle 100. Continuing the aboveexample, the regions are shown with three different priority levels forsensor detection. In particular, FIG. 5A presents a first view 500 ofthe rear seats of the vehicle and FIG. 5B presents a second view 510 ofthe rear seats. As shown, there are three types of areas, namely areas502 in dotted lines, areas 504 in dashed lines, and areas 506 indash-dot lines. A console 508 is also shown between the two rear seats.In this example, areas 502 are high priority, areas 504 are mediumpriority, and areas 506 are low priority.

FIG. 5C presents a view 520 of the backs of the front seats of avehicle. Here, areas 506 on the backs of the front seats are shown indash-dot lines. Also shown is a console 522, which is positioned betweenthe front seats. In various embodiments one or both consoles 508 and 522may be within the vehicle or not included at all. FIG. 5D presents afirst view 530 of the front seat section of the cabin, facing the seats.FIG. 5E presents a second view 540 of the front seat section from theviewpoint of the passenger(s) facing the front of the vehicle. And FIG.5F presents a view 550 of a trunk or other storage area of the vehicle(e.g., when the rear seats are folded down). Here, there may be a maincargo area encompassed by dash-dot region 504, and an entrance areaencompassed by dashed region 502. The entrance area may be, e.g., wherea passenger or other person leans in to place a package in the maincargo area. Coverage of this area may be very important, for instancewhen the vehicle determines whether to close the trunk.

Visual information detected in each area may be analyzed via an on-board(or remote) processing system for each of the regions. This may be done,for instance, using machine learning techniques to identify if a portionof the cabin differs from a baseline (“clean”) configuration. However,the ability to identify such differences may depend on the quality ofthe data, including sensor resolution. By way of example, one constraintfor an optical camera having resolution on the order of 1-4 megapixels(MP) is the ability to detect a matte black object of a predeterminedsize. For instance, the object may be between 70×40×5 mm and 110×90×20mm. While a higher resolution imager may be able to satisfactorilydetect an object smaller than the above examples, this may require anincreased amount of onboard processing power and/or data storage. Otherconstraints may be a low light signal to noise ratio (SNR) of between5:1 and 15:1 (or more or less), and a stray light rejection ratio on theorder of 0.8x⁻⁴ to 1.5x⁻⁴, for instance with a maximum source radius ofno more than ≤6.0°. The higher the stray light rejection ratio, thebetter the system is able to handle sources of light that could degradethe image quality, for instance due to image flare. This could come fromdashboard or window reflections, or from a user device (e.g., cellphoneflash turned on, etc.) The radius of the source represents how muchspace the stray light source is taking in the image.

The different regions and/or different priority levels of the cabin andstorage areas can be covered by a set of cameras distributed at variouslocations. FIG. 2 illustrated one example where cameras and othersensors 206, 208 and/or 210 may be located. In other examples, camerasand IR illuminators (and microphones) may be positioned in other areasof the interior, such as along a pillar (such as an A, B, C or Dpillar), in the console, in or adjacent to a door, in a headrest orarmrest, etc. Regardless of the exact position, the sensors should bearranged to provide fields of view sufficient to cover the selectedareas or zones that are of interest (e.g., the seats, floor, storageplaces). For example, narrower FOV cameras may be disposed in moreplaces in the vehicle or larger FOV in fewer places. In many scenarios,at least one camera is positioned per row of seating and one camera forthe trunk or other cargo space. For vehicles with an aisle, a pair ofcameras may be positioned on each side of the aisle per row. Microphonescan be positioned to capture in-vehicle audio. This may be used, forinstance, to assist the processing system when evaluating capturedimagery. Regardless of the placement of the cameras (and microphones),the components need to meet any applicable safety standards, includingFMVSS 201U (Occupant Protection in Interior Impact).

In one scenario, there is a least one camera arranged to view the frontseat areas, and at least one camera arranged to view the rear seatareas. Additional cameras may be arranged to provide sufficient coverageof the various cabin zones in view of the priority levels as notedabove. By way of example, the optical cameras may be fisheye lens-typecameras with IR illumination. For instance, the IR illumination may beon the order of 800 to 1000 nm.

According to one aspect, the IR illumination is able to cover the samefield of view as some or all of the cameras. In some examples, IRilluminators are collocated with respective cameras. And in otherexamples, the illumination devices are not collocated with the cameras.However, one constraint is that the IR illuminator(s) be placed so thatthe illumination does not degrade the image quality (due to flare). Forinstance, the IR illuminator(s) may be under the same constraint as forstray light rejection.

In some situations IR illumination can affect color accuracy. This couldpotentially affect the ability to correctly recognize a detected objectin an image. Thus, in one aspect the cameras may be calibrated tomaximize the color accuracy. The IR illumination for in-cabin camerasmay be activated based on ambient light conditions within the maincabin, whether the vehicle's headlights are on, and/or other criteria.For a camera(s) viewing the trunk area, there may be no IR illumination,IR illumination triggered according to ambient light conditions in thetrunk, or IR illumination only during a pre- or post-ride vehicle check.

Pre- and/or post-ride vehicle checks may be performed for the cabin areaand/or the trunk using the various cameras and illuminators. Additionalchecks may be performed during a ride (e.g., “mid-ride” checks). Howsuch mid-ride checks are performed may depend on time of day, drivingconditions, vehicle configurations and/or other factors. For example, atnight or other low-ambient light situations, interior visible cabinlights should not be turned on to perform a check, since this can bedistracting to the passenger(s) and could also degrade the imagery takenby the camera(s). Whether the vehicle has a solid roof, a moon roof or asunroof may also affect the imaging approach. In one example, theexposure time for a given check may be on the order of 50-150 ms (e.g.,100 ms) or more or less. An auto exposure algorithm may be used todefine the gain and exposure time for each camera. This will account forthe amount of ambient light and potential flaring from stray lightsources.

The interior sensors may be employed in various use situations, whichgenerally fall into pre-ride situations, post-ride situations, andmid-ride while transporting one or more passengers to theirdestination(s).

Imagery from the cameras (and potentially audio from the microphones)may be communicated to the on-board vehicle control system via a wiredlink (e.g., power over data line) or a wireless link. Then the systemmay perform real-time object recognition, for instance via machinelearning techniques, to identify different conditions or situations,such as a passenger being improperly buckled, an object has beeninadvertently left in the vehicle, cleaning is required, whether certainareas are clear (e.g., so that the trunk door can be closed or a windowrolled up), and whether someone in the driver's seat has their hands onthe steering wheel. Alternatively or additionally, certain imagery maybe sent to a remote assistance service for processing. For example, theremote assistance service may have more processing resources or a morerobust machine learning image processing technique that can identifyspecific items or other objects in the vehicle.

By way of example, in a pre-ride situation, the control system mayconfirm that the cabin is clean for the next rider and/or confirm thecabin is in a starting state condition (e.g. seats are in a nominalposition). If the cabin is not clear, the control system may contact aservice center to schedule a cleaning before the next pickup. If that isnot possible, prospective passengers may be alerted, ride fees may beadjusted, etc. If the seats are not in the starting state condition,then the control system may cause one or more actuators or other seatassemblies to adjust the seats accordingly, for instance by moving aseat forward or backward, adjusting the recline angle, headrest height,etc. Similar adjustments may be made for window or steering wheelpositioning, etc. Alternatively or additionally, such adjustment may beperformed based on user preferences of the passenger(s) to be picked upnext. FIGS. 6A-B illustrate one example of seat adjustment. As shown inFIG. 6A, the headrest of the front passenger seat may be fully extendedfrom a prior rider. The control system may lower the headrest to adefault height before the next pickup, as shown in FIG. 6B.

During a ride (e.g., “mid-ride”), as the vehicle is operating in anautonomous driving mode, the control system may confirm that riders areproperly seated and belted, confirm the number of riders based on thetrip request, determine seats used, and/or observe rider behavior suchas in response to certain driving operations, etc. By way of example,FIG. 7A illustrates a view 700. Based on obtained imagery, the controlsystem may determine that there are two passengers 702 a, 702 b in thefront seats, and that the seatbelts 704 a, 704 b are properly buckled.The system may also determine current seat configurations 706 a, 706 b,for instance to compare against the default seat positions pre-ride. Andas shown in 710 of FIG. 7B, the system may determine whether there areany packages 712 or other objects 714 located in the cabin (or thetrunk). This information can be used in the post-ride phase, for exampleto check that riders have exited the vehicle, check that the vehicle wasleft in a clean condition, and to check for items left behind. At theride's conclusion, the onboard system may remind passengers to take alltheir belongings with them. However, should something be left behind,the post-ride check identifies it. At this point, the vehicle may send amessage to the user's device regarding the left item. The message mayindicate when and where the item may be retrieved. Alternatively oradditionally, the vehicle may correspond with a remote assistanceservice to notify it about the left item. This can include scheduling astop at a service facility or other location so that the item may beremoved from the vehicle and stored until the user can retrieve it.Should the post-ride check indicate that the vehicle needs cleaning orother service (e.g., if the windows or headrests cannot be returned to adefault position), the vehicle may schedule a service or otherwiseadjust its planned route accordingly. Furthermore, the cameras may alsobe used to determine user sentiment or for other user experienceresearch. This can include using the cameras to learn what seats aremost preferred by passengers for a given vehicle or seatingconfiguration. This information may be employed when selecting anddispatching vehicles for rider pickups, to adjust camera placement andFOV, etc.

As explained above, the vehicle may communicate directly with thepassengers via its user interface system, or indirectly via itscommunication system. In the latter case, information may becommunicated to or received from an app on the customer's mobile phoneor other computing device. Such communication by the vehicle be doneduring the ride, as well as pre- and/or post-ride.

FIGS. 8A-B illustrate general examples of how information may becommunicated between the vehicle and the user.

In particular, FIGS. 8A and 8B are pictorial and functional diagrams,respectively, of an example system 800 that includes a plurality ofcomputing devices 802, 804, 806, 808 and a storage system 810 connectedvia a network 812. System 800 also includes vehicles 814, which may beconfigured the same as or similarly to vehicles 100 of FIGS. 1A-B.Vehicles 814 may be part of a fleet of vehicles. Although only a fewvehicles and computing devices are depicted for simplicity, a typicalsystem may include significantly more.

As shown in view 850 of FIG. 8B, each of computing devices 802, 804, 806and 808 may include one or more processors, memory, data andinstructions. Such processors, memories, data and instructions may beconfigured similarly to the ones described above with regard to FIG. 3.The various computing devices and vehicles may communication via one ormore networks, such as network 812. The network 812, and interveningnodes, may include various configurations and protocols including shortrange communication protocols such as Bluetooth™, Bluetooth LE™, theInternet, World Wide Web, intranets, virtual private networks, wide areanetworks, local networks, private networks using communication protocolsproprietary to one or more companies, Ethernet, WiFi and HTTP, andvarious combinations of the foregoing. Such communication may befacilitated by any device capable of transmitting data to and from othercomputing devices, such as modems and wireless interfaces.

In one example, computing device 802 may include one or more servercomputing devices having a plurality of computing devices, e.g., a loadbalanced server farm, that exchange information with different nodes ofa network for the purpose of receiving, processing and transmitting thedata to and from other computing devices. For instance, computing device802 may include one or more server computing devices that are capable ofcommunicating with the computing devices of vehicles 814, as well ascomputing devices 804, 806 and 808 via the network 812. For example,vehicles 814 may be a part of a fleet of vehicles that can be dispatchedby a server computing device to various locations. In this regard, thecomputing device 802 may function as a dispatching server computingsystem which can be used to dispatch vehicles to different locations inorder to pick up and drop off passengers or to pick up and delivercargo. In addition, server computing device 802 may use network 812 totransmit and present information to a user of one of the other computingdevices or a passenger of a vehicle. In this regard, computing devices804, 806 and 808 may be considered client computing devices.

As shown in FIG. 8A each client computing device 804, 806 and 808 may bea personal computing device intended for use by a respective user 816,and have all of the components normally used in connection with apersonal computing device including a one or more processors (e.g., acentral processing unit (CPU)), memory (e.g., RAM and internal harddrives) storing data and instructions, a display (e.g., a monitor havinga screen, a touch-screen, a projector, a television, or other devicesuch as a smart watch display that is operable to display information),and user input devices (e.g., a mouse, keyboard, touchscreen ormicrophone). The client computing devices may also include a camera forrecording video streams, speakers, a network interface device, and allof the components used for connecting these elements to one another. Asindicated in FIG. 8B, device 808 may be the device of a passenger who iscurrently in the vehicle, while device 806 may be the device of a riderawaiting pickup.

Although the client computing devices may each comprise a full-sizedpersonal computing device, they may alternatively comprise mobilecomputing devices capable of wirelessly exchanging data with a serverover a network such as the Internet. By way of example only, clientcomputing devices 806 and 808 may be mobile phones or devices such as awireless-enabled PDA, a tablet PC, a wearable computing device (e.g., asmartwatch), or a netbook that is capable of obtaining information viathe Internet or other networks.

In some examples, client computing device 804 may be a remote assistanceworkstation used by an administrator or operator to communicate withpassengers of dispatched vehicles, or users awaiting pickup. Althoughonly a single remote assistance workstation 804 is shown in FIGS. 8A-B,any number of such workstations may be included in a given system.Moreover, although workstation 804 is depicted as a desktop-typecomputer, the workstation 804 may include various types of personalcomputing devices such as laptops, netbooks, tablet computers, etc.

Storage system 810 can be of any type of computerized storage capable ofstoring information accessible by the server computing devices 802, suchas a hard-drive, memory card, ROM, RAM, DVD, CD-ROM, flash drive and/ortape drive. In addition, storage system 810 may include a distributedstorage system where data is stored on a plurality of different storagedevices which may be physically located at the same or differentgeographic locations. Storage system 810 may be connected to thecomputing devices via the network 812 as shown in FIGS. 8A-B, and/or maybe directly connected to or incorporated into any of the computingdevices. The remote assistance service or the server computing devicemay be used to provide enhanced image processing. For example, they mayhave more robust machine learning models than the onboard system, whichcan detect and categorize (classify) more types of objects. Based onthis categorization, the system may tailor the message to the passengeror other user about the item (e.g., an audible announcement that theuser's cell phone has fallen under the seat, or a message sent to an appon the user's device that a package has been left in the trunk and canbe picked up at a service center).

The vehicle or remote assistance may communicate directly or indirectlywith the passengers' client computing devices. Here, for example,information may be provided to the passengers regarding current drivingoperations, changes to the route in response to the situation, whetherto buckle up, caution about packages or other objects left in thevehicle, etc.

FIG. 9 illustrates a method of operation in accordance with theforegoing. In particular, it illustrates a flow diagram 900 of a methodof controlling an interior sensing system for a vehicle configured tooperate in an autonomous driving mode. At block 902, one or moreprocessors initiate image capture by the plurality of image sensors inaccordance with a predetermined ride checklist. The predetermined ridechecklist is stored in memory of a control system of the vehicle. Atblock 904, in response to the predetermined ride checklist, the one ormore processors receive captured imagery from a plurality of imagesensors disposed along different surfaces within an interior area of thevehicle. The plurality of image sensors is configured to capture imagesof the interior area of the vehicle. At block 906, the one or moreprocessors perform processing of the received imagery to detect acurrent status of the interior area of the vehicle based on detectedobjects or conditions within the vehicle. And at block 908, the one ormore processors cause one or more systems of the vehicle to perform anaction based on the current status of the interior area of the vehicle.

Finally, as noted above, the technology is applicable for various typesof vehicles, including passenger cars, buses, RVs and trucks or othercargo carrying vehicles.

Unless otherwise stated, the foregoing alternative examples are notmutually exclusive, but may be implemented in various combinations toachieve unique advantages. As these and other variations andcombinations of the features discussed above can be utilized withoutdeparting from the subject matter defined by the claims, the foregoingdescription of the embodiments should be taken by way of illustrationrather than by way of limitation of the subject matter defined by theclaims. In addition, the provision of the examples described herein, aswell as clauses phrased as “such as,” “including” and the like, shouldnot be interpreted as limiting the subject matter of the claims to thespecific examples; rather, the examples are intended to illustrate onlyone of many possible embodiments. Further, the same reference numbers indifferent drawings can identify the same or similar elements. Theprocesses or other operations may be performed in a different order orsimultaneously, unless expressly indicated otherwise herein.

1. An interior sensing system for a vehicle configured to operate in anautonomous driving mode, the interior sensing system comprising: aplurality of image sensors disposed within an interior area of thevehicle, the plurality of image sensors being configured to captureimages of the interior area of the vehicle; one or more infrared unitsconfigured to illuminate a set of zones within the vehicle during imagecapture by the plurality of image sensors; and a control systemoperatively coupled to the plurality of image sensors and the one ormore infrared units, the control system including one or more processorsconfigured to: determine an amount of ambient light; actuate the one ormore infrared units in response to the determined amount of ambientlight; initiate image capture by the plurality of image sensors inconjunction with actuation of the one or more infrared units; detect, inresponse to the image capture, a current status of the interior area ofthe vehicle based on one or more detected objects or conditions withinthe vehicle; and based on the current status of the interior area of thevehicle, either (i) cause a system of the vehicle to perform an actionor (ii) or determine how to present information to at least onepassenger.
 2. The interior sensing system of claim 1, wherein thedetermination of how to present information to the at least onepassenger is based on detection of a proximity of the at least onepassenger to one or more display devices within a passenger compartmentof the vehicle.
 3. The interior sensing system of claim 1, wherein thedetermination of how to present information to the at least onepassenger is based on either a position or a line of sight of the atleast one passenger to one or more display devices within a passengercompartment of the vehicle.
 4. The interior sensing system of claim 1,wherein the information is information regarding current drivingoperations or a change to the route.
 5. The interior sensing system ofclaim 1, wherein the information is information regarding whether tobuckle up or a caution about an object left in the vehicle.
 6. Theinterior sensing system of claim 1, wherein the set of zones comprises aplurality of zones, at least first and second ones of the plurality ofzones having different priority levels.
 7. The interior sensing systemof claim 6, wherein at least one of an image capture rate or an imageresolution in each of the first and second zones is based on thepriority level corresponding to that zone.
 8. The interior sensingsystem of claim 6, wherein a field of view for a selected one of theplurality of image sensors focused on a given one of the plurality ofzones is based on the priority level for the given zone.
 9. The interiorsensing system of claim 6, wherein an image processing technique usedfor captured imagery is based on the priority level for a given one ofthe plurality of zones.
 10. The interior sensing system of claim 6,wherein the one or more processors of the control system are furtherconfigured to request remote assistance when either (i) the prioritylevel for a given one of the plurality of zones exceeds a threshold, or(ii) there is uncertainty about captured imagery.
 11. The interiorsensing system of claim 1, wherein the one or more processors of thecontrol system are configured to initiate the image capture by theplurality of image sensors in conjunction with actuation of the one ormore infrared units to maximize a color accuracy of one or more of theimage sensors.
 12. The interior sensing system of claim 1, wherein theone or more processors of the control system are configured to initiatethe image capture by the plurality of image sensors in conjunction withactuation of the one or more infrared units based whether the vehicle'sheadlights are on.
 13. The interior sensing system of claim 1, whereininitiation of image capture by the plurality of image sensors inconjunction with actuation of the one or more infrared units isperformed according to a ride check operation.
 14. A method ofcontrolling an interior sensing system for a vehicle configured tooperate in an autonomous driving mode, the method comprising:determining an amount of ambient light; actuating, by one or moreprocessors, one or more infrared units in response to the determinedamount of ambient light, the one or more infrared units configured toilluminate a set of zones in an interior area of the vehicle;initiating, by the one or more processors, image capture by a pluralityof image sensors in conjunction with actuating the one or more infraredunits, the plurality of image sensors being configured to capture imagesof the interior area of the vehicle; detecting by the one or moreprocessors, in response to the image capture, a current status of theinterior area of the vehicle based on one or more detected objects orconditions within the vehicle; and based on the current status of theinterior area of the vehicle, the one or more processors either (i)causing a system of the vehicle to perform an action or (ii) ordetermining how to present information to at least one passenger. 15.The method of claim 14, wherein determining how to present informationto the at least one passenger is: based on detection of a proximity ofthe at least one passenger to one or more display devices within apassenger compartment of the vehicle; based on a position the at leastone passenger to the one or more display devices; or based on a line ofsight of the at least one passenger to the one or more display devices.16. The method of claim 14, wherein the information is: informationregarding current driving operations; information regarding a change tothe route; information regarding whether to buckle up; or informationregarding a caution about an object left in the vehicle.
 17. The methodof claim 14, wherein the set of zones comprises a plurality of zones, atleast first and second ones of the plurality of zones having differentpriority levels.
 18. The method of claim 17, wherein: at least one of animage capture rate or an image resolution in each of the first andsecond zones is based on the priority level corresponding to that zone;or a field of view for a selected one of the plurality of image sensorsfocused on a given one of the plurality of zones is based on thepriority level for the given zone.
 19. The method of claim 17, furthercomprising requesting remote assistance when either (i) the prioritylevel for a given one of the plurality of zones exceeds a threshold, or(ii) there is uncertainty about captured imagery.
 20. The method ofclaim 14, wherein initiating the image capture by the plurality of imagesensors in conjunction with actuation of the one or more infrared unitsis: done to maximize a color accuracy of one or more of the imagesensors; based whether the vehicle's headlights are on; or performedaccording to a ride check operation.